Go Basic & Around project

Project Introduction

Around: a Geo-index based social network

• Built a scalable web service in Go to handle posts and deployed to Google Cloud (GAE flex) for better scaling

• Utilized ElasticSearch (GCE) to provide geo-location based search functions such that users can search nearby posts within a distance (e.g. 200km)

• Used Google Dataflow to implement a daily dump of posts to BigQuery table for offline analysis

• Aggregated the data at the post level and user level to improve the keyword based spam detection (BigQuery)

constants.js

export const API_ROOT = 'https://around-75015.appspot.com';
export const TOKEN_KEY = 'TOKEN';
export const AUTH_PREFIX = 'Bearer';

Why we need to learn about Go?

Answer: server language for next generation (both computer friendly and developer friendly)

Who is using Golang

• Google (Creator), Uber, AirBnb
• Dropbox, Facebook, eBay, Heroku, Douban, Jingdong, Meituan

As opposed to Java, Go is compiled to machine code and is executed directly. Much like C.

Go example

f, err := os.Open("filename.ext")
    if err != nil {
        log.Fatal(err)
    }

Other benefits

• Multiple return values
• Multi-threading
  • Go Routine
  • Channels
• Has borrowed many good ideas from python

Python is really easy to break.

• Does not compile. You will never know it breaks until it breaks.
• Language is confusing.
• Good for testing and evaluation with many excellent machine learning libraries.

>>> a = set(['hippo'])
>>> a
set(['hippo'])
>>> a = set('hippo')
>>> a
set(['h', 'i', 'p', 'o'])

Setup Local Environment (Go)

Make sure you have installed Go based on Prerequisite.
Mac Users (Windows User wait a minute)

Step 1.1.1 Open Terminal, enter

go version

It should show something like go version go1.10.2 darwin/amd64.

Step 1.1.2(Optional: GOPATH default to ~/go) In the same Terminal, enter

mkdir ~/go
echo export GOPATH=~/go >> ~/.bash_profile
source ~/.bash_profile
echo $GOPATH

In here we must not overwrite the bin of go, add export PATH=$PATH:/usr/local/go/bin to .bash_profile and update zsh profile

Step 1.1.3 In the same Terminal, enter

mkdir -p  ~/Documents/Around/service
cd  ~/Documents/Around/service
vim main.go

Step 1.1.4 Type ‘a’ or ‘i’ and then copy a hello world program into main.go

package main

import "fmt"

func main() {
	fmt.Println("Hello, world")
}

Step 1.1.5 Exit with :wq , in the terminal, type

go run main.go

You should be able to see the output of “Hello, world”

First Go program

First, we need to define some objects in a Go program to represent the data we store.

Step 1.3.1 let’s define the struct for post and location.

Encode json object (https://golang.org/pkg/encoding/json/)

package main

import "fmt"

type Location struct {
      Lat float64 `json:"lat"`
      Lon float64 `json:"lon"`
}

type Post struct {
      // `json:"user"` is for the json parsing of this User field. Otherwise, by default it's 'User'.
      User     string `json:"user"`
      Message  string  `json:"message"`
      Location Location `json:"location"`
}

func main() {
      fmt.Println("Hello, world")
}

Step 1.3.2 Add one method handlerPost() after main() to handle Post.

func main() {
      fmt.Println("Hello, world")
}

func handlerPost(w http.ResponseWriter, r *http.Request) {
      // Parse from body of request to get a json object.
      fmt.Println("Received one post request")
      decoder := json.NewDecoder(r.Body)
      var p Post
      if err := decoder.Decode(&p); err != nil {
             panic(err)
             return
      }

      fmt.Fprintf(w, "Post received: %s\n", p.Message)
}

In here if you add rawString json:"lat" after variable, it will parse it automatically

What does this method do? If user sends a http request with a body as

{
  “user”: “jack”
  “message”: “this is a message”
}

Then it will automatically construct a Post object named p and update its values to be p.User = “jack” and p.Message = “this is a message”

Fmt %s means string

Just one line of code to decode json object into go object. In comparison, if you do it in java.

JSONObject venue = getVenue(event);
      if (venue != null) {
        if (!venue.isNull("address")) {
          JSONObject address = venue.getJSONObject("address");
          StringBuilder sb = new StringBuilder();
          if (!address.isNull("line1")) {

            sb.append(address.getString("line1"));
          }
          if (!address.isNull("line2")) {
            sb.append(address.getString("line2"));
          }
          if (!address.isNull("line3")) {
            sb.append(address.getString("line3"));
          }
          builder.setAddress(sb.toString());
        }
        if (!venue.isNull("city")) {
          JSONObject city = venue.getJSONObject("city");
          builder.setCity(getStringFieldOrNull(city, "name"));
        }
        if (!venue.isNull("country")) {
          JSONObject country = venue.getJSONObject("country");
          builder.setCountry(getStringFieldOrNull(country, "name"));
        }
        if (!venue.isNull("state")) {
          JSONObject state = venue.getJSONObject("state");
          builder.setState(getStringFieldOrNull(state, "name"));
        }
        builder.setZipcode(getStringFieldOrNull(venue, "postalCode"));
        if (!venue.isNull("location")) {
          JSONObject location = venue.getJSONObject("location");
          builder.setLatitude(getNumericFieldOrNull(location, "latitude"));
          builder.setLongitude(getNumericFieldOrNull(location, "longitude"));
        }
      }

Step 1.3.3 Replace main function to call handlerPost when started. Final version:

import (
      "fmt"
      "net/http"
      "encoding/json"
      "log"
)

func main() {
      fmt.Println("started-service")
      http.HandleFunc("/post", handlerPost)
      log.Fatal(http.ListenAndServe(":8080", nil))
}

Step 1.3.5 Open your postman,
Choose ‘POST’, the url is http://localhost:8080/post, in the request body, enter

{
	"user":"1111",
	"message":"一生必去的100个地方",
	"location":{
		"lat":37.5,
		"lon":-120.1
	}
}

Click ‘Send’, you should be able to see a response with

Post received: 一生必去的100个地方

Add another handler for search (called it handlerSearch), the request has a url pattern like
http://localhost:8080/search?lat=10.0&lon=20.0. Parse it and then print out the lat and lon.

Note: to get request parameters from url

lat := r.URL.Query().Get("lat")

Step 1.3.6 Answer

func main() {
      fmt.Println("started-service")
      http.HandleFunc("/post", handlerPost)
      http.HandleFunc("/search", handlerSearch)
      log.Fatal(http.ListenAndServe(":8080", nil))
}

func handlerSearch(w http.ResponseWriter, r *http.Request) {
      fmt.Println("Received one request for search")
      lat := r.URL.Query().Get("lat")
      lon := r.URL.Query().Get("lon")

      fmt.Fprintf(w, "Search received: %s %s", lat, lon)
}

Step 1.3.7 return a JSON object. Change handlerSearch to be

Import (
      …
      "strconv"
)

const (
      DISTANCE = "200km"
)

func handlerSearch(w http.ResponseWriter, r *http.Request) {
      fmt.Println("Received one request for search")
      lat, _ := strconv.ParseFloat(r.URL.Query().Get("lat"), 64)
      lon, _ := strconv.ParseFloat(r.URL.Query().Get("lon"), 64)
      // range is optional 
      ran := DISTANCE 
      if val := r.URL.Query().Get("range"); val != "" { 
         ran = val + "km" 
      }

      fmt.Println("range is ", ran)

      // Return a fake post
      p := &Post{
             User:"1111",
             Message:"一生必去的100个地方",
             Location: Location{
                    Lat:lat,
                    Lon:lon,
             },
      }

      js, err := json.Marshal(p)
      if err != nil {
             panic(err)
             return
      }

      w.Header().Set("Content-Type", "application/json")
      w.Write(js)
}

在Go里面，变量必须得用到，不然会报错，除非用下划线做变量名

Step 1.3.8 Open Postman, url is http://localhost:8080/search?lat=10.0&lon=20.0&range=300 click send again. You should see something like this

{
    "user": "1111",
    "message": "一生必去的100个地方",
    "location": {
        "lat": 10,
        "lon": 20
    }
}